• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    Various example embodiments relate to acquiring image streams from which image data is analyzed in two phases in order to provide information about attention displayed information receives from viewers, while personally identifiable information is not permanently stored. An apparatus, a method, and a computer program are disclosed.
    公开号:FI20205453A1
    申请号:FI20205453
    申请日:2020-05-04
    公开日:2021-11-05
    发明作者:Mika Suominen；Ville Santeri Vanhakartano；Ilkka Samuli Laine
    申请人:Lainetech Oy；
    IPC主号:
    专利说明:

    [0001] [0001] Various example embodiments generally relate to the field of information technology. In particular, some example embodiments relate to measuring and analyzing attention of information using machine vision.BACKGROUND
    [0002] [0002] Especially in urban areas there are a lot of displays, billboards, and display windows providing information to passers-by, for example, for advertising purposes. For example, a location and content of the information may affect to a degree of attention the presented information receives. It would be beneficial to be able to provide data about the received attention to further analyze impact of the information.SUMMARY
    [0003] [0003] This summary 1s provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or o 25 essential features of the claimed subject matter, nor is O it intended to be used to limit the scope of the claimed 3 subject matter. S [0004] Example embodiments provide an apparatus to E provide information about attention displayed @ 30 information receives from viewers. The information about 3 received attention is obtained by a two-phase analysis O of image data, where personally identifiable information is not permanently stored. These benefits may be achieved by the features of the independent claims. Further implementation forms are provided in the dependent claims, the description, and the drawings.
    [0005] [0005] According to a first aspect, there is provided an apparatus. The apparatus is configured to acquire image stream from at least one imaging device configured to monitor area in front of displayed information; analyze the image stream to detect at least one face indicative of a viewer of the displayed information in an image frame of the image stream; store the image frame to a memory in response to detecting the at least one face; analyze the image frame to detect each face in the image frame; analyze the image frame to identify a plurality of facial attributes of each face in the image frame; store the facial attributes of each face as a text record of the viewer to a database; delete the image frame from the memory; and provide the records stored in the database for a user, wherein the records indicate attention of the displayed information received by viewers.
    [0006] [0006] In an embodiment, the plurality of facial attributes comprises at least one of a gender, an age range, an emotional state or an indication whether eyes are open or closed.
    [0007] [0007] In an embodiment, in addition or o 25 alternatively, the image frame comprises at least one of S a timestamp or a location of the displayed information. 3 [0008] In an embodiment, in addition or S alternatively, each image frame comprises an imaging E device-specific identifier and the apparatus is @ 30 configured to store location information of at least one 3 of the imaging device or the displayed information O associated with the imaging device-specific identifier in the database; and the location information corresponding to the imaging device-specific identifier of the image frame is stored in the text record.
    [0009] [0009] In an embodiment, in addition or alternatively, the apparatus is further configured to analyze the image frame to detect eyes in response to detecting the at least one face in the image frame; and wherein the image frame is stored in response to detecting both the face and the eyes.
    [0010] [0010] In an embodiment, the apparatus is configured to analyze the image stream using a Haar feature-based cascade classifier.
    [0011] [0011] In an embodiment, the apparatus is configured to compare the facial attributes of stored image frames with adjacent timestamps, wherein one of the image frames is used as a referenceto identify the same face in a plurality of image frames based on similarity of the facial attributes; and wherein the facial attributes of the same face in the plurality of image frames are stored in the text record only for the reference image frame.
    [0012] [0012] In an embodiment, in addition or alternatively, to determine a length of gaze based on the same face identified in the plurality of image frames with adjacent timestamps by comparing the timestamps of the first and the last image frame of the plurality of > 25 image frames; and the length of gaze is stored in the N text record. S [0013] In an embodiment the apparatus is further S configured to acguire the image stream in real time, and E analyze the image stream in real time. 2 30 [0014] According to a second aspect, there is 3 provided a method. The method comprises acquiring image S stream from at least one imaging device configured to monitor area in front of displayed information;
    [0015] [0015] According to a third aspect, there is provided a computer program comprising a program code which, when executed by at least one processing unit, causes the at least one processing unit to perform the method of the second aspect.
    [0016] [0016] In an embodiment, the computer program is embodied on a computer readable medium.
    [0017] [0017] Many of the attendant features will be more readily appreciated as they become better understood by reference to the following detailed description considered in connection with the accompanying drawings. 9 25 < DESCRIPTION OF THE DRAWINGS 3 [001 8] The accompanying drawings, which are included S to provide a further understanding of the example E embodiments and constitute a part of this specification, @ 30 illustrate example embodiments and together with the 3 description help to understand the example embodiments. O In the drawings:
    [0019] [0019] FIG. 1 illustrates an example of an apparatus to provide data on received attention according to an example embodiment.
    [0020] [0020] FIG. 2 illustrates an example configuration 5 for generating data on received attention to a user according to an example embodiment.
    [0021] [0021] FIG. 3 illustrates an example arrangement of detection units for detecting received attention according to an example embodiment.
    [0022] [0022] FIG. 4 illustrates a flow chart of operation of an apparatus analyzing received attention according to an example embodiment.
    [0023] [0023] Like references are used to designate like parts in the accompanying drawings.DETAILED DESCRIPTION
    [0024] [0024] Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The detailed description provided below in connection with the appended drawings is intended as a description of the present examples and is not intended to represent the only forms in which the present example may be constructed or utilized. The description sets forth the functions of the example and N 25 the seguence of operations for constructing and N operating the example. However, the same or equivalent S functions and seguences may be accomplished by different S examples. j [0025] According to an example embodiment, an O 30 apparatus may comprise means for acguiring image stream, S for example a real time image stream. The apparatus may N be configured to acauire image stream from a plurality of imaging devices installed towards an area in front of a displayed information.
    [0027] [0027] The image stream may be received in real time or it may not be real time. For example, there is a delay in stream or the stream is stored, buffered, transmitted or processed later.
    [0028] [0028] FIG. 1 illustrates an example embodiment of an apparatus 100 to provide data on received attention according to an example embodiment. The apparatus 100 may comprise at least one processor 101. The at least one processor 101 may comprise, for example, one or more of various processing devices, such as for example a co- processor, a microprocessor, a controller, a digital o 25 signal processor (DSP), a processing circuitry with or O without an accompanying DSP, or various other processing ro devices including integrated circuits such as, for x example, an application specific integrated circuit z (ASIC), a field programmable gate array (FPGA), a N 30 microcontroller unit (MCU), a hardware accelerator, a = special-purpose computer chip, or the like. ä [0029] The apparatus 100 may further comprise at least one memory 102, such as a local memory. The memory 102 may be configured to store, for example, computer program code or the like, for example operating system software and application software. The memory 102 may comprise one or more volatile memory devices, one or more non-volatile memory devices, and/or a combination thereof. For example, the memory may be embodied as magnetic storage devices (such as hard disk drives, magnetic tapes, etc.), optical magnetic storage devices, or semiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.).
    [0030] [0030] The apparatus 100 may further comprise a transceiver 104 configured to enable the apparatus 100 to transmit and/or receive information, to/from other apparatuses. The transceiver 104 may be configured to IS provide at least one wireless radio connection, such as for example a 3GPP mobile broadband connection (e.g. 3G, 4G, 5G). However, the transceiver 104 may be configured to provide one or more other type of connections, for example a wireless local area network (WLAN) connection such as for example standardized by IEEE 802.11 series or Wi-Fi alliance; a short range wireless network connection such as for example a Bluetooth; a wired connection such as for example a local area network (LAN) connection, a universal serial bus (USB) connection or o 25 an optical network connection, or the like; or a wired O Internet connection. ro [0031] When the apparatus 100 is configured to x implement some functionality, some component and/or z components of the apparatus 100, such as for example the N 30 at least one processor 101 and/or the memory 102, may be = configured to implement this functionality. Furthermore, N when the at least one processor 101 is configured to N implement some functionality, this functionality may be implemented using program code 103 comprised, for example, in the memory 102.
    [0032] [0032] The functionality described herein may be performed, at least in part, by one or more computer program product components such as software components.
    [0033] [0033] The apparatus 100 comprises means for performing at least one method described herein. In one example, the means comprises the at least one processor 101, the at least one memory 102 including program code 103 configured to, when executed by the at least one o 25 processor 101, cause the apparatus 100 to perform the O method. 3 [0034] The apparatus 100 may comprise one or more 3 imaging devices 105. The imaging devices 105 may be E configured to provide continuous image stream in real- Ro 30 time when powered on. The imaging device 105 may D comprise, for example, a high-resolution camera. The S image resolution of the camera may be sufficient for N analysis by machine vision applications, for example, to distinguish faces and eyes. The required resolution may depend on the size of a monitored area and how far away the area which the imaging device 105 monitors is located at. In an embodiment the camera may be 360 degrees camera having 360 degree view.
    [0035] [0035] The apparatus 100 may comprise for example a computing device such as for example a server or a database unit 106. In an embodiment, the apparatus 100 comprises an image processing unit configured to provide image frames for analysis to the database unit 106. Alternatively, the image frames may be analyzed by the image processing unit and the output of the analysis may be provided to the database unit 106 by the image processing unit in a text format. The database unit 106 may be configured to store records comprising information from analysis of the image frames to a database in a text format. The database unit 106 may be configured to provide an interface for queries from a user. Although the apparatus 100 is illustrated as a single device it is appreciated that, wherever applicable, functions of the apparatus 100 may be distributed to a plurality of devices, for example to implement example embodiments as a cloud computing service.
    [0036] [0036] FIG. 2 illustrates an example configuration o 25 for generating data on received attention to a user O according to an example embodiment. In an embodiment, an ro apparatus may comprise one or more detection devices x 200. The detection device 200 may comprise at least one I imaging device 202 configured to provide real-time image N 30 stream. The detection device 200 may further comprise an = image processing unit 201. The image processing unit 201 N may be configured to receive the image stream and analyze N each image frame in real-time to detect at least one face. In response to at least one detected face, the image processing unit 201 may store the image frame locally to a memory.
    [0037] [0037] FIG. 3 illustrates an example arrangement of imaging devices 301, 302 for detecting received attention according to an example embodiment.
    [0039] [0039] In the first part of the analysis, image stream may be captured by one or more imaging devices at 402. The image stream may be received at 403 by an image processing unit configured to analyze the image stream, for example in real-time, at 404. In an embodiment, the image processing unit may comprise an algorithm for object detection, such as a Haar-feature based cascade classifier. The algorithm may be trained with a lot of images with faces and images without faces. Then, Haar features comprising kernels may be used, wherein each feature may be a single value obtained by subtracting sum of pixels under a white rectangle from sum of pixels under a black rectangle. Haar feature is only an example o 25 embodiment and other kind of algorithm for object O detection may be used in the embodiments. A plurality of ro possible sizes and locations of each kernel may be used x to calculate thousands of features. In order to reduce = the number of calculations of the features, integral N 30 image may be used such that the calculations for a given = pixel may be reduced to an operation involving only four N pixels. Further, the most relevant features may be N selected from the plurality of features. For example, the first feature may focus on the property that the region of the eyes is often darker (i.e. black rectangle) than the region of the nose and cheeks (i.e. white rectangle). The second feature may focus on the property that the eyes are darker than the bridge of the nose.
    [0041] [0041] At 407, the database unit 401 may carry out facial analysis for the received image frames stored at
    [0042] [0042] In an embodiment, the timestamped image frames o 25 may be compared with each other such that one image frame O is selected as a reference image frame. The reference ro image frame may be compared to image frames with adjacent x timestamps to measure a similarity score against the I reference image frame. If the comparison results in N 30 substantially same values for face attributes, i.e. a = high similarity score, it may be deduced that the N detected face in the image frames with adjacent N timestamps and high similarity score belongs to the same person. Hence, only one database record for the same person may be formed instead of a plurality of records corresponding to the same viewer. In an embodiment, timestamps of the image frames with the same viewer may be used to estimate length of gaze of the viewer towards the displayed information. For example, timestamp of the first and the last image frame of the plurality of image frames with adjacent timestamps may be used to calculate an estimation of the length of the gaze for the identified same viewer. The obtained length of gaze may be stored in the record of the reference image frame.
    [0043] [0043] At 409, the record may be stored to the database. The database may comprise records of analyzed image frames from a plurality of imaging devices. The database may be used by a user to search for information based on different search conditions, such as the location, time, and individual or a set of face attributes. In addition, the database may be used by the user, for example, to create statistics. In an embodiment, the database unit 401 may be configured to provide the user an access to the database via a customer portal. For example, the user may retrieve statistic about attention received by information displayed by the user to gain valuable information about the reached audience and their reactions. For example, the user may o 25 auery statistics about the attention displayed O information received at 75th Maple Street between 1st May ro and 25th May. The database may return a response x comprising, for example: "5023 viewers; 78% amused; 93% I females; 55% teenagers; ..”. N 30 [0044] At 410, the stored image frame is deleted. = Fach image frame may be deleted, for example, in response N to the completed facial analysis and stored record by N the database unit 401. Hence, no data which may enable identifying individuals is permanently stored. No already analyzed image data remains either at the image processing unit 400 or the database unit 401 after the stored image frame is deleted at 410.
    [0045] [0045] Further features of the method (s) directly result for example from functionalities of the apparatus and its components such as the imaging device, image processing unit and database unit described throughout the specification and in the appended claims and are therefore not repeated here. Different variations of the method (s) may be also applied, as described in connection with the various example embodiments.
    [0046] [0046] An apparatus may be configured to perform or cause performance of any aspect of the method(s) described herein. Further, a computer program may comprise instructions for causing, when executed, an apparatus to perform any aspect of the method(s) described herein. Further, an apparatus may comprise means for performing any aspect of the method(s) described herein. According to an example embodiment, the means comprises at least one processor, and memory including program code, the at least one processor, and program code configured to, when executed by the at least one processor, cause performance of any aspect of the method (s). o 25 [0047] Any range or device value given herein may be S extended or altered without losing the effect sought. 3 Also, any embodiment may be combined with another 3 embodiment unless explicitly disallowed. E [0048] Although the subject matter has been described @ 30 in language specific to structural features and/or acts, 3 it is to be understood that the subject matter defined O in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as examples of implementing the claims and other eguivalent features and acts are intended to be within the scope of the claims.
    [0049] [0049] It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to those that solve any or all of the stated problems or those that have any or all of the stated benefits and advantages. It will further be understood that reference to 'an' item may refer to one or more of those items.
    [0050] [0050] The operations of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate. Additionally, individual blocks may be deleted from any of the methods without departing from the scope of the subject matter described herein. Aspects of any of the embodiments described above may be combined with aspects of any of the other embodiments described to form further embodiments without losing the effect sought.
    [0051] [0051] The term 'comprising' is used herein to mean including the method, blocks, or elements identified, but that such blocks or elements do not comprise an exclusive list and a method or apparatus may contain S 25 additional blocks or elements. N [0052] Although subjects may be referred to as 3 ‘first’ or ‘second’ subjects, this does not necessarily S indicate any order or importance of the subjects. E Instead, such attributes may be used solely for the @ 30 purpose of making a difference between subjects. 3 [0053] It will be understood that the above O description is given by way of example only and that various modifications may be made by those skilled in the art. The above specification, examples and data provide a complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity, or with reference to one or more individual embodiments, those skilled in the art could make numerous alterations to the disclosed embodiments without departing from scope of this specification. oONON
    LÖ <Q + o
    I a aO
    LO +
    LO oONON
    权利要求:
    Claims (12)
    [1] 1. An apparatus, configured to: acquire image stream from at least one imaging device configured to monitor area in front of displayed information; analyze the image stream to detect at least one face indicative of a viewer of the displayed information in an image frame of the image stream; store the image frame to a memory in response to detecting the at least one face; analyze the image frame to detect each face in the image frame; analyze the image frame to identify a plurality of facial attributes of each face in the image frame; store the facial attributes of each face as a text record of the viewer to a database; delete the image frame from the memory; and provide the records stored in the database for a user, wherein the records indicate attention of the displayed information received by viewers.
    [2] 2. The apparatus of claim 1, wherein the plurality of facial attributes comprises at least one of a gender, an age range, an emotional state or an indication whether S eyes are open or closed. 3 <
    [3] 3. The apparatus of claim 1 or 2, wherein the image 2 frame comprises at least one of a timestamp or a location N 30 of the displayed information. = S
    [4] 4. The apparatus of claim 1 or 2, wherein each image N frame comprises an imaging device-specific identifier and the apparatus is configured to store location information of at least one of the imaging device or the displayed information associated with the imaging device-specific identifier in the database; and the location information corresponding to the imaging device-specific identifier of the image frame is stored in the text record.
    [5] 5. The apparatus of any preceding claim, wherein the apparatus is further configured to analyze the image frame to detect eyes in response to detecting the at least one face in the image frame; and wherein the image frame is stored in response to detecting both the face and the eyes.
    [6] 6. The apparatus of any preceding claim, wherein the apparatus is configured to analyze the image stream using a Haar feature-based cascade classifier.
    [7] 7. The apparatus of claim 3, wherein the apparatus is configured to compare the facial attributes of stored image frames with adjacent timestamps, wherein one of the image frames is used as a reference to identify the same face in a plurality of image frames based on similarity of the facial attributes; and wherein the facial attributes of the same face in S the plurality of image frames are stored in the text ro record only for the reference image frame. 3 I
    [8] 8. The apparatus of claim 7, wherein the apparatus > 30 is configured to determine a length of gaze based on the O same face identified in the plurality of image frames S with adjacent timestamps by comparing the timestamps of N the first and the last image frame of the plurality of image frames; and the length of gaze is stored in the text record.
    [9] 9. The apparatus of any preceding claim, the apparatus being further configured to acquire the image stream in real time; and analyze the image stream in real time.
    [10] 10. A method, comprising: acquiring image stream from at least one imaging device configured to monitor area in front of displayed information; analyzing the image stream to detect at least one face indicative of a viewer of the displayed information in an image frame of the image stream; storing the image frame to a memory in response to detecting the at least one face; analyzing the image frame to detect each face in the image frame; analyzing the image frame to identify a plurality of facial attributes of each face in the image frame; storing the facial attributes of each face as a text record of the viewer to a database; deleting the image frame from the memory; and providing the records stored in the database for a user, wherein the records indicate attention of the S displayed information received by viewers. 3 <
    [11] 11. A computer program comprising a program code 2 which, when executed by at least one processing unit, > 30 causes the at least one processing unit to perform the E method of claim 10.
    S N
    [12] 12. A program according to claim 11, wherein the computer program 1s embodied on a computer readable medium.
    oO
    N
    O
    N
    LÖ
    I + o
    I a a
    O
    LO +
    LO oO
    N
    O
    N
    类似技术:
    公开号 | 公开日 | 专利标题
    US9560323B2|2017-01-31|Method and system for metadata extraction from master-slave cameras tracking system
    US10204264B1|2019-02-12|Systems and methods for dynamically scoring implicit user interaction
    US8582816B2|2013-11-12|Method and apparatus for video analytics based object counting
    US7949157B2|2011-05-24|Interpreting sign language gestures
    JP2011210238A|2011-10-20|Advertisement effect measuring device and computer program
    US9245247B2|2016-01-26|Queue analysis
    WO2021143228A1|2021-07-22|Data pushing method and apparatus, electronic device, computer storage medium and computer program
    CN108229289B|2021-02-05|Target retrieval method and device and electronic equipment
    US8947600B2|2015-02-03|Methods, systems, and computer-readable media for detecting scene changes in a video
    WO2020114116A1|2020-06-11|Pedestrian detection method based on dense crowds, and storage medium and processor
    Sirmacek et al.2011|Automatic crowd analysis from airborne images
    FI20205453A1|2021-11-05|An apparatus and a method to measure attention received by displayed information
    Pavlov et al.2013|Application for video analysis based on machine learning and computer vision algorithms
    JP2011070629A5|2012-11-29|
    JP2011070629A|2011-04-07|Advertising effect measurement system and advertising effect measurement device
    Bouma et al.2013|WPSS: Watching people security services
    US10922531B2|2021-02-16|Face recognition method
    KR101124560B1|2012-03-16|Automatic object processing method in movie and authoring apparatus for object service
    US20200073877A1|2020-03-05|Video cookies
    Meessen et al.2006|Content-based retrieval of video surveillance scenes
    US11144763B2|2021-10-12|Information processing apparatus, image display method, and non-transitory computer-readable storage medium for display control
    US20210350138A1|2021-11-11|Method to identify affiliates in video data
    CN109977824B|2021-06-01|Article taking and placing identification method, device and equipment
    WO2021135138A1|2021-07-08|Target motion trajectory construction method and device, and computer storage medium
    Fritz et al.2010|Semantic analysis of human visual attention in mobile eye tracking applications
    同族专利:
    公开号 | 公开日
    FI20205453A|2021-11-05|
    WO2021224546A1|2021-11-11|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US20080147488A1|2006-10-20|2008-06-19|Tunick James A|System and method for monitoring viewer attention with respect to a display and determining associated charges|
    WO2011031932A1|2009-09-10|2011-03-17|Home Box Office, Inc.|Media control and analysis based on audience actions and reactions|
    法律状态:
    优先权:
    申请号 | 申请日 | 专利标题
    FI20205453A|FI20205453A|2020-05-04|2020-05-04|An apparatus and a method to measure attention received by displayed information|FI20205453A| FI20205453A|2020-05-04|2020-05-04|An apparatus and a method to measure attention received by displayed information|
    PCT/FI2021/050316| WO2021224546A1|2020-05-04|2021-04-28|An apparatus and a method to measure attention received by displayed information|
    [返回顶部]